Multiplexing and demultiplexing haptic signals

ABSTRACT

A system receives a multiplexed signal with two or more different types of haptic signals encoded therein. Each type of haptic signal represents a haptic effect for different types of haptic output devices. The system determines a target haptic output device located on a haptic playback device. The system demultiplexes the multiplexed signal into at least the type of haptic signal corresponding to the target output device. The system provides the demultiplexed haptic signal to the target haptic output device.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority of U.S. patent application Ser. No.14/185,102, filed on Feb. 20, 2014 and issued as U.S. Pat. No.9,317,120, which claims priority of Provisional Patent Application Ser.No. 61/907,138, filed on Nov. 21, 2013, and Provisional PatentApplication Ser. No. 61/874,920, filed on Sep. 6, 2013. The contents ofeach of these applications are hereby incorporated by reference.

FIELD

One embodiment is directed to a haptically-enabled device. Moreparticularly, one embodiment is directed to a system that multiplexes ordemultiplexes haptic signals to generate haptic effects.

BACKGROUND INFORMATION

Electronic device manufacturers strive to produce a rich interface forusers. Conventional devices use visual and auditory cues to providefeedback to a user. In some interface devices, kinesthetic feedback(such as active and resistive force feedback) and/or tactile feedback(such as vibration, texture, and heat) is also provided to the user,more generally known collectively as “haptic feedback” or “hapticeffects.” Haptic feedback can provide cues that enhance and simplify theuser interface. For example, vibration effects, or vibrotactile hapticeffects, may be useful in providing cues to users of electronic devicesto alert the user to specific events, or provide realistic feedback tocreate greater sensory immersion within a simulated or virtualenvironment.

Haptic feedback has also been increasingly incorporated in portable andmobile electronic devices, such as cellular telephones, smartphones,portable gaming devices, vehicle based devices and interfaces, and avariety of other portable and mobile electronic devices. For example,some portable gaming applications are capable of vibrating in a mannersimilar to control devices (e.g., joysticks, etc.) used withlarger-scale gaming systems that are configured to provide hapticfeedback.

In order to generate vibration or other effects, many devices utilizesome type of actuator or haptic output device. Known actuators used forthis purpose include an electromagnetic actuator such as an solenoidactuator, an Eccentric Rotating Mass (“ERM”) actuator in which aneccentric mass is moved by a motor, a Linear Resonant Actuator vibrationmotor (“LRA”), or a piezoelectric actuator. Each of these target hapticactuators receives a haptic control signal that provides the parametersfor the haptic effect. However, for a particular haptic effect, thehaptic control signal may need to be varied based on the actuator thatprovides the haptic effect. Thus, for one haptic actuator, the controlsignal may be one type of signal representing, for example, an on or offcondition, while for another haptic actuator, the control signal may beanother type of signal representing, for example, a position value.

SUMMARY

One embodiment receives a multiplexed signal with two or more differenttypes of haptic signals encoded therein. Each type of haptic signalrepresents a haptic effect for different types of haptic output devices.The system determines a target haptic output device located on a hapticplayback device. The system demultiplexes the multiplexed signal into atleast the type of haptic signal corresponding to the target outputdevice. The system provides the demultiplexed haptic signal to thetarget haptic output device.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a haptically-enabled system in accordance withone embodiment of the present invention.

FIG. 2 is a block diagram illustrating high level logic multiplexinghaptic control signals in accordance with one embodiment.

FIG. 3 is a flow diagram illustrating multiplexing and demultiplexinghaptic control signals in accordance with one embodiment.

FIG. 4 is a flow diagram illustrating multiplexing and demultiplexinghaptic control signals in accordance with one embodiment.

FIG. 5 is a flow diagram illustrating multiplexing and demultiplexinghaptic control signals in accordance with one embodiment

FIG. 6 is a flow diagram illustrating multiplexing and demultiplexinghaptic control signals in accordance with one embodiment.

FIG. 7 is a block diagram illustrating high level logic multiplexinghaptic control signals into a multi-channel stream in accordance withone embodiment.

FIG. 8 contains an example of header information for a multi-channelfile in accordance with one embodiment.

FIG. 9 is a flow diagram illustrating multiplexing and demultiplexinghaptic control signals in accordance with one embodiment.

DETAILED DESCRIPTION

One embodiment is a system that generates at least two haptic signalsfrom the same input or source. The two haptic signals represent a hapticeffect to be played on two different target haptic output devices. Thesystem multiplexes/encodes the signals into a single multiplexed hapticsignal stream sent to a haptic playback device. At the haptic playbackdevice, the signals are de-multiplexed/decoded and the appropriatehaptic signal is played on the device based on the haptic output devicetype.

FIG. 1 is a block diagram of a haptically-enabled system 10 inaccordance with one embodiment of the present invention. System 10includes a touch sensitive surface 11 or other type of user interfacemounted within a housing 15, and may include mechanical keys/buttons 13.Internal to system 10 is a haptic feedback system that generatesvibrations on system 10. In one embodiment, the vibrations are generatedon touch surface 11.

The haptic feedback system includes a processor or controller 12.Coupled to processor 12 is a memory 20 and an actuator drive circuit 16,which is coupled to an actuator 18. Actuator 18 can be any type ofDirect Current (“DC”) motor, including without limitation an EccentricRotating Mass (“ERM”), a Linear Resonant Actuator vibration motor(“LRA”), a piezoelectric motor, or a solenoid actuator. In addition toor in place of actuator 18, system 10 may include other types of hapticoutput devices (not shown) that may be non-mechanical or non-vibratorydevices such as devices that use electrostatic friction (“ESF”),ultrasonic surface friction (“USF”), devices that induce acousticradiation pressure with an ultrasonic haptic transducer, devices thatuse a haptic substrate and a flexible or deformable surface or shapechanging devices and that may be attached to a user's body, devices thatprovide projected haptic output such as a puff of air using an air jet,devices that provide electrical muscle stimulation, etc.

Processor 12 may be any type of general purpose processor, or could be aprocessor specifically designed to provide haptic effects, such as anapplication-specific integrated circuit (“ASIC”). Processor 12 may bethe same processor that operates the entire system 10, or may be aseparate processor. Processor 12 can decide what haptic effects are tobe played and the order in which the effects are played based on highlevel parameters. In general, the high level parameters that define aparticular haptic effect include magnitude, frequency, and duration. Lowlevel parameters such as streaming motor commands could also be used todetermine a particular haptic effect. A haptic effect may be considered“dynamic” if it includes some variation of these parameters when thehaptic effect is generated or a variation of these parameters based on auser's interaction.

Processor 12 outputs the control signals to actuator drive circuit 16,which includes electronic components and circuitry used to supplyactuator 18 with the required electrical current and voltage (i.e.,“motor signals”) to cause the desired haptic effects. In instances wherethe haptic effects correspond to the playback of a multimedia file, suchas a video file, processor 12 may provide the haptic control signal tothe haptic drive circuit. System 10 may include more than one actuator18, and each actuator may include a separate drive circuit 16, allcoupled to a common processor 12. Memory device 20 can be any type ofstorage device or computer-readable medium, such as random access memory(“RAM”) or read-only memory (“ROM”). Memory 20 stores instructionsexecuted by processor 12. Among the instructions, memory 20 includes ahaptic effects module 22 which are instructions that, when executed byprocessor 12, generate drive signals for actuator 18 that provide hapticeffects, as disclosed in more detail below. Memory 20 may also belocated internal to processor 12, or any combination of internal andexternal memory.

Touch surface 11 recognizes touches, and may also recognize the positionand magnitude of touches on the surface. The data corresponding to thetouches is sent to processor 12, or another processor within system 10,and processor 12 interprets the touches and in response generates hapticeffect signals. Touch surface 11 may sense touches using any sensingtechnology, including capacitive sensing, resistive sensing, surfaceacoustic wave sensing, pressure sensing, optical sensing, etc. Touchsurface 11 may sense multi-touch contacts and may be capable ofdistinguishing multiple touches that occur at the same time. Touchsurface 11 may be a touchscreen that generates and displays images forthe user to interact with, such as keys, dials, etc., or may be atouchpad with minimal or no images.

System 10 may be a handheld device, such a cellular telephone, personaldigital assistant (“PDA”), smartphone, computer tablet, gaming console,vehicle based interface, etc., or may be any other type of device thatincludes a haptic effect system that includes one or more actuators. Theuser interface may be a touch sensitive surface, or can be any othertype of user interface such as a mouse, touchpad, mini-joystick, scrollwheel, trackball, game pads or game controllers, etc. In embodimentswith more than one actuator, each actuator may have a differentrotational capability in order to create a wide range of haptic effectson the device.

In addition to providing user interfacing haptic effects, system 10 mayprovide statically generated haptic effects for playback in system 10along with, for example, a video or audio file.

Different devices support different levels of haptic playbackcapability. The type of haptic signals supported by a haptic playbackdevice depends on a combination of the actuator used and the drivecircuit of the actuator. Some devices may only support “basic haptics”which control a haptic actuator such as actuator 18 with a haptic signalwhich turns the actuator on or off according to an on/off binary signal.One sample rate for a basic haptic signal can be 200 Hz, but othersample rates may be used. In one example of a basic haptic signal, anERM actuator may be configured to respond to a basic haptic signal. Ifthe control signal tells the ERM to be on, the haptic drive circuit(e.g., drive circuit 16) for an ERM actuator will turn the ERM actuatoron in accordance with the basic haptic signal.

Some devices may support “standard definition” haptics which control ahaptic actuator such as actuator 18 with a haptic signal that varies theintensity of the haptic effect according to a signal encoded with one of128 non-negative values for each sample of the standard definitionhaptic signal. One sample rate for a standard haptic signal can be 200Hz, but other sample rates may be used. In one example of a standarddefinition haptic signal, an LRA actuator may be configured to respondto a standard definition haptic signal. A haptic drive circuit for anLRA actuator will control the LRA actuator according to the amplitudeinformation at each 5 ms sample.

Some devices may support “high definition” haptics which control ahaptic actuator such as actuator 18 with a haptic signal that varies theintensity of the haptic effect according to a signal encoded with avalue of +/−127 for each sample of the high definition haptic signal.One sample rate for a high definition haptic signal can be 8 kHz, butother sample rates may be used. High definition haptic signals typicallyhave higher sample rates than basic haptic or standard definition hapticsignals. In one example of a high definition haptic signal, apiezoelectric actuator may be configured to respond to a high definitionhaptic signal. The haptic drive circuit for a piezoelectric actuator maycontrol the piezoelectric actuator in accordance with the amplitudeinformation every 125 μs.

Furthermore, the ramp-up/down times vary significantly between LRAactuators and ERM actuators and even between different models of ERMactuators. This means that even though the same type of signal may beused for multiple ERMs, a haptic effect programmer may want to specifydifferent effects based on each actuator's capabilities. A haptic effectprogrammer may want to provide a customized haptic signal for amultitude of target devices with each signal being produced from thesame source.

A problem arises when trying to deliver a single piece of media contentthat plays reasonably well across the widest range of devices.Therefore, one issue that haptic effect programmers face is that theymay need to account for providing multiple haptic control signals basedon the type of actuator or other haptic output device in the playbackdevice. In addition, programmers may not know which type of hapticoutput device will be in use on the playback device. To deal with thisproblem, programmers could provide a high definition or standarddefinition signal intended for a piezoelectric actuator or LRA actuatorand allow the end device to “warp” that signal into lesser or greaterdefinition signals as required. This is not ideal because, for example,in the case of a standard definition haptic signal intended for an LRAactuator but actually playing back on a piezoelectric actuator,granularity would be lost in converting a 200 Hz signal into an 8 kHzsignal, basically turning a piezoelectric actuator into an expensive LRAactuator. In the case of a high definition signal intended for apiezoelectric actuator but actually playing back on an ERM actuatorsupporting only basic haptics, the conversion may render the ERMactuator ineffectual.

One option may be to provide a haptic control signal for each of theexpected range of haptic output devices in the end device, but in doingso, the bandwidth requirements for the haptic signals would be increasedand some delivery mechanism is needed to organize the signals. Anotheroption may be to encode two or more haptic signals together in a lossyway to incorporate data from the two or more haptic signals into onesignal using the normally allotted bandwidth for the highest bitratesignal. Both of these approaches are contemplated herein.

Encode/Decode Multiplexing

Some embodiments multiplex multiple haptic signals so that each retainsits native representation without requiring an end device, such asdevice 10, to convert one format into another format, and withoutincreasing the bandwidth required to provide the haptic control signals.These embodiments provide at least an endpoint solution for a basichaptic signal; because capabilities of an end device are not necessarilyknown to a haptic programmer ahead of time, without at least a basichaptic signal available, content play within applications running onplayback devices can be severely hamstrung by inadequate haptic signalconversion results. By multiplexing a basic haptic signal into thecombined haptic waveform, at least a high quality basic hapticexperience is produced.

FIG. 2 is a block diagram illustrating high level logic multiplexinghaptic control signals in accordance with one embodiment. Elements 205,210, and 215 correspond to haptic streams A, B, and C, respectively. At220, haptic streams A, B, and C are multiplexed into a single combinedstream. At 225, the combined stream is delivered to the playback devicefor rendering into haptic effects. At 230, the combined stream isdemultiplexed at the playback device such as device 10 of FIG. 1 forpresentation to the user. At 240, haptic stream A is considered atprocessor 12 and if the stream type for stream A corresponds to the typeof actuator used in the system, at 244, the stream is played on thesystem by drive circuit 16 and actuator 18, otherwise at 242, the streamis ignored. At 250, haptic stream B is considered at processor 12 and ifthe stream type for stream B corresponds to the type of actuator used inthe system, at 254, the stream is played on the system by drive circuit16 and actuator 18, otherwise at 252, the stream is ignored. At 260,haptic stream C is considered at processor 12 and if the stream type forstream C corresponds to the type of actuator used in the system, at 264,the stream is played on the system by drive circuit 16 and actuator 18,otherwise at 262, the stream is ignored. Alternatively, in someembodiments, demultiplexer 230 has access to information describing thetype of actuator used in system 10. In such embodiments, atdemultiplexing 230, the demultiplexer extracts and sends the appropriatehaptic stream to drive circuit 16.

In some embodiments, multiplexing is performed within a hapticprogramming environment by a haptic programmer or by using a tool thatcan analyze audio files, such as audio files associated with a videofile, and generate a haptic effects file to correspond to the audiofile. For example, Avid Corporation's Pro Tools “AudioSuite” plugin cancontain an off-line audio-to-haptics converter. Conversion algorithmsmay also be used in batch command line formats to convert audio signalsto haptic signals and can also be used to multiplex the signals asdescribed herein. Three basic types haptic effects files can begenerated: a basic haptic signal, a standard definition haptic signal,and a high definition haptic signal (these three are also referred toherein as “control signals”). One of skill in the art will understandthat additional types may be developed and that other parameters may beinvolved in the conversion so that each type can have multiplevariations of haptic streams. Whether the source is an audio sourceconversion or haptic effects specified by a haptic programmer, each ofthese three haptic streams are derived from the same input or source(and are intended to provide the same general haptic effect), but differdue to the different haptic output device technologies available. Thesethree streams can be multiplexed at 220 into a single stream. Themultiplexed signal can then be downloaded or streamed to haptic playbackdevice 10. The multiplexing and demultiplexing can be done in differentdevices or the same device. The multiplexed signal will be demultiplexedat 230 in playback device 10 and the haptic signal corresponding to thetype of actuator 18 in playback device 10 will be used to play thehaptic effects. The other streams can be ignored. In some embodiments,demultiplexing at 230 may occur at a streaming server, with playbackdevice 10 requesting a particular haptic signal based on the type ofactuator 18. The streaming server can then demultiplex the hapticsignals and stream the appropriate haptic signal for actuator 18.

FIG. 3 is a flow diagram illustrating multiplexing and demultiplexinghaptic control signals in accordance with one embodiment. In oneembodiment, the functionality of the flow diagram of FIG. 3 (and FIGS.4-6 and 9 below) is implemented by software stored in memory or othercomputer readable or tangible medium, and executed by a processor. Inother embodiments, the functionality may be performed by hardware (e.g.,through the use of an application specific integrated circuit (“ASIC”),a programmable gate array (“PGA”), a field programmable gate array(“FPGA”), etc.), or any combination of hardware and software.

In some embodiments, a basic haptic signal is multiplexed with astandard definition signal. As described above, the basic haptic signalis generally made up of discrete samples of single bits of informationrepresenting an on or off state at a frequency of about 5 ms or 200 Hz.The standard definition control signal is generally made up of discreteamplitude samples of a signed byte of information also at a frequency of200 Hz. The multiplexing process in FIG. 3 in one embodiment is abit-field combination where the sign bit of the standard definitionsignal is used to encode the basic haptic signal instead.

At 305, a basic haptic sample and standard definition sample arereceived from a basic haptic stream and standard definition stream,respectively. At 310, the lower 7 bits of the standard definition signalare extracted by bit wise AND-ing the byte with the value 0x7F(01111111). The resulting byte will always lead with a 0 (zero). In someembodiments, if the standard definition signal is a full range signalusing values of +/−127, the sample byte may be right shifted to dividethe value by two, but preserve the sign bit. At 315, the basic hapticsignal is represented by a byte (either 0x00 or 0x01, in hexadecimal)and the bits are shifted left 7 times, to move the least significant bitinto the most significant bit, resulting in either a value of 0x80 or0x00. At 320, the standard definition and basic haptic signals arecombined by bit wise OR-ing the two values to produce a single byte (orthey could be added together), with the most significant bitcorresponding to the basic haptic signal and the least significant bitscorresponding to the standard definition signal. This is repeated forevery sample to produce the entire multiplexed haptic signal.

To demultiplex the combined signal at playback time, at 350, thecombined signal would be received as a single byte of information for asample. At 355, the standard definition signal sample would be extractedby bit-wise AND-ing the byte with 0x7F to produce a standard definitionsample byte. In some embodiments, if the standard definition signal is afull range signal using original values of +/−127, the signal sample maybe left shifted to multiply the sample value by two, restoring the signbit. Interpolation, explained in further detail below, may be used torestore granularity between signal levels that was lost when the fullrange standard definition signal was divided by two. At 360, the basichaptic signal would be extracted by taking the original combined signaland right shifting 7 times, putting the most significant bit into theleast significant bit's place and filling with leading zeros. At 365,the proper signal is chosen for the actuator type found in the device.The appropriate demultiplexed signal is sent to actuator 18 by processor12.

FIG. 4 is a flow diagram illustrating multiplexing and demultiplexinghaptic control signals in accordance with one embodiment. For thisembodiment, a high definition haptic signal (used for example, with apiezoelectric actuator) is multiplexed with a standard definition hapticsignal. In some embodiments, a high definition haptic signal isrepresented by a full waveform at 8 kHz with a value at any given samplebeing a byte of information with a value ranging from −127 to 127. Otherembodiments may represent a high definition haptic signal by a fullwaveform at other frequencies and ranges, such as at a frequency of 16kHz. In some embodiments, the frequency of a high definition signal canvary. In some embodiments, a standard definition control signal (usedfor example, with an LRA actuator) is an amplitude signal at 200 Hzrepresented by a signed byte of information at any given sample. Otherembodiments may represent a standard definition signal at otherfrequencies and ranges, such as at a frequency of 300 Hz. In someembodiments, the frequency of a standard definition signal can alsovary. One of ordinary skill will understand that these frequencies andranges may vary, but will understand that an actuator designed to play ahigh definition signal can typically handle a much higher frequencysignal than an actuator designed to play a standard definition signal.The multiplexing process in FIG. 4 can create a DC offset over samplesubsets of the high definition signal.

At 405, the high definition and standard definition control signals arereceived. At 410, the standard definition control signal is up-sampledto match the high definition control signal frequency. Someinterpolation to reduce stair stepping and provide smoothing may bedesired depending on whether the standard definition signal frequency isa direct multiple of the high definition control frequency. At 415, thetwo signals are divided by two, losing the remainder. This division maybe performed by right-shifting each sample one place, losing the leastsignificant bit. In other embodiments, at 415, the two signals arescaled such that their addition falls within the expected range of theoutput. For example, in the case where the output is expected to rangefrom +/−127, in one embodiment, the high definition signal may be scaledby 60% and the standard definition signal may be scaled by 40%, suchthat their addition will equal 100%. This can be accomplished by knownmeans, for example, by a multiplier with a float value or by a dividerwith a scaled value. The values may be scaled at any representationdepending on how the designer wishes to emphasize the accuracy of onesignal over the other. Generally, the higher scaled signal will be moreaccurate when decoding/demultiplexing. At 420, the two signals are addedtogether to produce a combined signal. For some signals, thiseffectively creates a DC offset to the high definition signal becausethe standard definition signal changes at a much lower frequency thanthe high definition signal and so appears as a DC offset to the highdefinition control signal. Because the two signals were divided by two(or scaled to some other value as described above), some granularity islost for both signals because the range was reduced from −127 through127 to −63 through 63. This is repeated for every sample to produce theentire multiplexed haptic signal.

To demultiplex the combined signal at playback device 10, at 450, thecombined multiplexed signal is received. Any given sample of the 8 kHzcombined signal is represented by a byte. At 455, the DC offset isremoved. Techniques for removing a DC offset are known in the art. Forthe standard definition signal, the combined signal can be put through alow-pass filter, allowing the low frequency standard definition signalto pass through, but preventing the high frequency high definitionsignal from passing through. For the high definition signal, thestandard definition signal can be subtracted from the combined signal toproduce the high definition signal. Alternatively, the combined signalcan be put through a high-pass filter, allowing the high frequency highdefinition signal to pass through, but preventing the low frequencystandard definition signal from passing through. Other techniques thatmay be used to remove the DC offset include an enveloping techniquewhich is similar to the low-pass filter in this example. At 460, each ofthe extracted signals for standard definition control and highdefinition control are multiplied by two (or scaled back to theiroriginal proportional values as described above, e.g., divided by 60% inthe case of the high definition signal) to bring them back to theapproximate values as before multiplexing. In one embodiment, to improvegranularity, interpolation may be used between values as desired toprovide smoother transitions. For example, a set of samples with thevalues 4, 4, 6, 6, 8, 8, 10 after demultiplexing and doubling could beinterpolated to be 4, 5, 6, 7, 8, 9, 10. Also at 460, the standarddefinition signal can be down-sampled back to 200 Hz or whatever thestarting frequency was. At 465, the proper signal is chosen for theactuator type found in device 10. The appropriate demultiplexed signalis sent to the actuator 18 by processor 12.

In some embodiments, the multiplexing and demultiplexing can account forthe characteristics of the effective frequencies of the standarddefinition and high definition signals. Even though the high definitionsignal is sampled at a greater frequency than a standard definitionsignal, the standard definition signal may be up-sampled to match thehigh definition signal. These signals may be combined by taking theaverage of the two signals, sample by sample (divide each sample by 2and add them). The act of dividing the signals by 2 reduces theresolution of each by one bit (an 8-bit per sample signal becomes a7-bit per sample signal). This loss of resolution causes a slightreduction in the fidelity of both signals. After the two signals arecombined, the up-sampled standard definition signal is similar to a DCoffset to the high definition signal. To extract the standard definitionsignal, a low pass filter can be used to isolate the DC offset. A notchor high-pass filter could be used to isolate the frequencies of the highdefinition signal.

It is possible that the high definition signal, prior to multiplexingcould contain a DC offset. In this case, adding in the standarddefinition signal may increase or negate the DC offset of themultiplexed signal, which after demultiplexing, would result in loss ofinformation in the high definition signal at low frequencies anddistortion in the standard definition signal from the inclusion of lowfrequency information from the high definition signal. Thus, in someembodiments the flow above may be altered so that at 415, prior toadding the two signals together, filtering the high definition signalwith a high pass filter when the standard definition signal frequency isfairly low, for example less than 30 Hz, or notch filtering the highdefinition signal with a notch filter when the standard definitionvaries more, for example a frequency of 175 Hz to 250 Hz. Then on thedemultiplexing side, at 455, in the case where the high pass filter wasapplied to the high definition signal at 415, a low pass filter can beapplied to the multiplexed signal to extract the standard definitionsignal. In the case where the notch filter was applied to the highdefinition signal at 415, a band-pass filter can be used to extract thestandard definition signal. Because the standard definition signal canfill in the filtered spots in the high definition signal, themultiplexed signal can be used for the high definition signal. Asmoothing filter can be used on the high definition signal.

In addition, the two multiplexing techniques in FIGS. 3 and 4 may becombined to provide a three-way multiplexed signal that includes a basichaptic signal, a standard definition haptic signal, and a highdefinition haptic signal. To multiplex the signals, first the basichaptic signal would be multiplexed with the standard definition hapticsignal as described in conjunction with FIG. 3, and then the multiplexedbasic/standard definition signal would function as the standarddefinition signal for multiplexing with the high definition signal asdescribed above in conjunction with FIG. 4. To demultiplex, first thebasic/standard definition signal would be demultiplexed from the highdefinition as described in conjunction with FIG. 4. Next, the standarddefinition and basic haptic signals would be demultiplexed as describedabove in conjunction with FIG. 3.

FIG. 5 is a flow diagram illustrating multiplexing and demultiplexinghaptic control signals in accordance with one embodiment. As analternative to the multiplexing technique in FIG. 4, rather thancombining the standard definition and high definition signals to producea signal combined for every sampled byte, the two signals may beinterlaced. At 505, the standard definition and high definition hapticcontrol signals are received. At 510, the signals are combined bysubstituting samples of the standard definition signal into the highdefinition signal. Because the standard definition signal occurs at amuch lower frequency as compared to the high definition signal, thesample for the high definition signal can be substituted with the samplefor the standard definition signal without much loss to the highdefinition signal. For example, if the high definition signal is at 8kHz and the standard definition signal is at 200 Hz, then a standarddefinition sample occurs once for every 40 high definition samples. Thusthe 40th high definition sample can be substituted with thecorresponding standard definition sample. This, of course, causes somesamples of the piezoelectric signal to be lost.

In some embodiments, the standard definition signal can also beup-sampled to allow the substitution to occur more often. For example,the standard definition signal could be effectively up-sampled to 1 kHzby substituting every eighth high definition sample with the standarddefinition sample and repeating the same substitution five times withsame value standard definition sample. In some embodiments, when thestandard definition signal frequency is fairly high, for example afrequency of 175 Hz to 250 Hz, the envelope of the standard definitionsignal can be extracted using known techniques and up-sampled which willpreserve the haptic intention and content. Then, a new standarddefinition signal can be created with a higher sampling rate (like 1kHz, as above) and the signal multiplied by the envelope. Increasing thesubstitution frequency may help by providing for smaller processingbuffers to pass the high definition and standard definition controlsignals through to the actuator as appropriate. Also, increasing thesubstitution frequency increases the granularity for which the standarddefinition samples are processed and may provide better performance.However, increasing the substitution frequency also increases the lossof signal information in the high definition signal. No matter whatsubstitution frequency is chosen, in some embodiments, loss can beminimized by encoding the high definition signal value sampleimmediately prior to the substituted value to replace the mostsignificant bits of the prior sample with the least significant bits ofthe removed sample. This retains some of the sample information for theremoved sample by encoding it into the previous sample.

To demultiplex the combined standard definition and high definitioncontrol signals, at 550 the multiplexed signal is received. At 555, themultiplexed signal is subsampled at the same frequency for whichsubstitution was used. In the case where the substitution occurred at200 Hz (the same as the standard definition signal frequency), thiswould be every 5 ms. The subsamples can be combined to represent thestandard definition signal and the remaining samples can be combined torepresent the high definition signal. At 560, the remaining signal isprocessed for the high definition signal. Removing the standarddefinition signal leaves a hole in the high definition signal where thestandard definition signal was substituted. The hole can be filled bytaking an average of the signal value before and after the hole, byrepeating the signal value before the hole, or by repeating the signalvalue after the hole. In some embodiments, if the previous sample's fourmost significant bits were encoded with the missing sample information'sfour least significant bits as described above, then the missing samplecan be decoded by taking the four most significant bits of the previoussample as the four least significant bits of the missing sample andborrowing the four most significant bits of the sample preceding theprevious sample for both the previous sample and the missing sample. Inone embodiment, the four most significant bits of the missing sample canbe borrowed from the sample immediately following the missing sample. Inother words, considering a sequence of samples, the four mostsignificant bits for the missing sample would come from the sample tothe right and the four most significant bits for the previous samplewould come from the sample to the left. This encoding/decoding techniqueimproves performance because the most significant bits are unlikely tochange from one sample to the next, but change more gradually over manysamples.

In addition, the two multiplexing techniques in FIGS. 3 and 5 may becombined to provide a three-way multiplexed signal that includes a basichaptic signal, a standard definition haptic signal, and a highdefinition haptic signal. To multiplex the signals, first the basichaptic signal would be multiplexed with the standard definition hapticsignal as described in conjunction with FIG. 3, and then the multiplexedbasic/standard definition signal would function as the standarddefinition signal for multiplexing with the high definition signal asdescribed in conjunction with FIG. 5. To demultiplex, first thebasic/standard definition signal would be demultiplexed from the highdefinition signal as described in conjunction with FIG. 5. Next, thestandard definition and basic haptic signals would be demultiplexed asdescribed above in conjunction with FIG. 3.

FIG. 6 is a flow diagram illustrating multiplexing and demultiplexinghaptic control signals in accordance with one embodiment. Amplitudemodulation can be used to multiplex all available haptic signals,including multiple alternative basic haptic, standard definition, andhigh definition signals. At 605, all the haptic signals are received. At610, a sinusoidal carrier wave is selected for each of the signals suchthat none of the selected carrier waves are octaves (multiples) of eachother. At 615, a time ordered amplitude signal is generated for each ofthe haptic signals. At 620, each of the amplitude signals is added toeach of the selected carrier waves. At 625, each of the carrierwave/amplitude signals are combined together to produce the multiplexedsignal. The data size of the signal depends on the frequency waves andvalid range values selected for the carrier waves. For example, if thecombined carrier waves allowed for a multiplexed value for any givensample to be greater than an 8-bit number, then extra bits may need tobe added to the combined signal, thereby increasing the size andcomplexity. Further, the combined carrier wave should be sampled at thelargest frequency of the carrier waves and the haptic signal data. Thus,the data size is dependent on the selection of the carrier waves and mayvary accordingly.

To demultiplex the combined amplitude modulated haptic signal sources,at 650, the multiplexed signal is received. At 655, the multiplexedsignal is multiplied by the carrier wave in the proper phase for thesignal for the source haptic signal desired. At 660, the resultingsignal is square rooted to find the source haptic signal. At 665, ifadditional haptic signals need to be found in the combined wave, go backto 655. This is repeated for each of the carrier waves used to find allof the haptic source signals. If, at the receiving end, only one of thehaptic source signals is needed, then only the corresponding carrierwave needs to be processed against the modulated signal. At 670, theappropriate haptic signals are chosen based on the type of actuator inplayback device 10.

Channel Multiplexing

In other embodiments, providing multiple haptic signals to a hapticplayback device can be accomplished by combining all of the hapticsignals into a multi-channel playback stream. In embodiments thatimplement channel multiplexing, the types of signals, such as a basichaptic, standard definition, and high definition signals described aboveremain the same types of signals that can be used in channelmultiplexing. In addition, the haptic streams can be created usinghaptic programming and audio conversion tools, as described above.

In one embodiment, a haptic elementary stream (“HES”) is created byproviding header information to describe the content of a multi-channelstream that can provide two or more haptic streams. Such streams aresimilar to a stereo audio stream with a right track and a left track,except that each HES can hold up to 255 streams by implementation. Oneof ordinary skill can readily change the limit of the number ofsupported streams either up or down by increasing or decreasing theaddress space of the channel identifiers.

FIG. 7 is a block diagram illustrating high level logic multiplexinghaptic control signals into a multi-channel stream in accordance withone embodiment. Elements 705, 710, and 715 correspond to haptic streamsA, B, and C, respectively. At 720, haptic streams A, B, and C arecombined into a single multi-channel stream. At 725, the multi-channelstream is delivered to the playback device for rendering into hapticeffects. In some embodiments, the multi-channel stream is delivered to amultimedia server that can select the proper haptic stream for theplayback device and provide only that stream to the playback device,typically along with audio or video. At 730, the combined stream isdemultiplexed into individual streams from the multi-channel stream atthe playback device such as device 10 of FIG. 1 (or server) forpresentation to the user. At 740, haptic stream A is considered atprocessor 12 and if the stream type for stream A corresponds to the typeof actuator used in the system, at 744, the stream is played on thesystem by drive circuit 16 and actuator 18, otherwise at 742, the streamis ignored. At 750, haptic stream B is considered at processor 12 and ifthe stream type for stream B corresponds to the type of actuator used inthe system, at 754, the stream is played on the system by drive circuit16 and actuator 18, otherwise at 752, the stream is ignored. At 760,haptic stream C is considered at processor 12 and if the stream type forstream C corresponds to the type of actuator used in the system, at 764,the stream is played on the system by drive circuit 16 and actuator 18,otherwise at 762, the stream is ignored. In some embodiments,demultiplexer 730 can have access to information describing the type ofactuator used in system 10. In such embodiments, at demultiplexing 730,the demultiplexer extracts and sends the appropriate haptic stream todrive circuit 16.

Each sample of the multi-channel stream can contain one byte for eachchannel. For example, if there are two streams in the multi-channelstream, there are two channels, with a total of 16-bits per sample. Forhaptic streams that are larger than 8 bits per sample, the number ofbits used by each channel can be increased accordingly. In someembodiments, the number of bits can be rounded up to the nearestbyte-length or nibble-length. For example, if a haptic signal has asample with a value taking 10 bits, it could be stored in a channel ofthe HES that is 16 bits if rounding by byte-lengths or 12-bits ifrounding by nibble-lengths.

Information about how the HES file is organized can be contained in aheader. The HES file header can contain information describing thecontent of the stream such as, the number of channels, the number ofbits per sample, the sample rate, an indicator to indicate which channelcorresponds to which type of haptic signal or intended haptic playbackdevice type, encoding information, compression information, and versioninformation.

FIG. 8 contains an example of header information for a multi-channelfile in accordance with one embodiment. Each slot of the lines of 16slots represents a nibble; two consecutive slots represent a byte. At805, the HES file is contained in a standard Resource Interchange FileFormat (“RIFF”) file envelope. The HES file can consist of a definedheader format, as in 810, and a defined data portion, as in 815. Inheader 810, the byte for 820 represents a major version identifier toidentify the major version associated with the HES file. The byte for825 represents a minor version identifier to identify the minor versionassociated with the HES file. For example, if the HES version were 6.2,the 6 can correspond to the major version and the 2 can correspond tothe minor version. The byte for 830 represents encoding info for thedata in file which can include values corresponding to encoding schemessuch as “Uncompressed LPCM,” “Time-Ordered Amplitudes,” and the like.The byte for 835 can be reserved for some future use associated withfuture versions. The four bytes for 840 can represent the sampling ratein hertz. The two bytes for 845 can represent the number of bits persample. The byte for 850 can represent the number of channels in thefile. The bytes for 855, 860, and 865 can describe a particular actuatoror end device types for each channel. The number of bytes reserved todescribe each channel should correspond to the number of channels, onefor each channel. Thus, the byte for 855 can specify the target hapticdevice for channel 1; the byte for 860 can specify the target hapticdevice for channel 2; and the byte for 865 can specify the target hapticdevice for channel N. The byte for 870 can represent a compressionscheme put on the data. For example, known lossy and lossless datacompression schemes can be applied to the data.

For HES data 815, an illustrative example is shown that specifies twochannels for each sample, with 8-bits for each channel. The two bytesfor 875 can represent the first sample, and the first two nibbles canrepresent the first channel haptic stream while the second two nibblescan represent the second channel haptic stream. The two bytes for 880are set up the same as 875 but can have different values. The patterncan repeat for all samples until the last sample, sample N, representedby the two bytes 885. One of ordinary skill will recognize that thenumber of nibbles or bytes used for each of the fields of HES header 810can be changed at will in accordance with a particular preference.Further, one of ordinary skill will recognize that the format of HESdata 815 will change based on the parameters established in the headerportion, which can increase the number of channels per sample and thenumber of bits per sample.

FIG. 9 is a flow diagram illustrating multiplexing and demultiplexinghaptic control signals in accordance with one embodiment. Formulti-channel multiplexing, at 905, the haptic signals are received. Asnoted above, these signals can be generated from the same source and cangenerally represent the same haptic effects but on different targethaptic playback devices. In some embodiments, a multi-channel signal cancontain channels also for representing other haptic effects. Forexample, a multi-channel haptic signal can contain three channels forone haptic effect from one source and three channels for another hapticeffect from another source. In such cases, the target haptic playbackdevice may have multiple haptic output devices and each sourcecorresponds to one haptic output device. At 910, the received signalsare analyzed and up-sampled to the highest frequency signal. Also at910, the samples are padded to produce a uniform width sub-sample foreach channel for each sample of the multi-channel stream. For example, abasic haptic signal can be up-sampled to match the sampling frequency ofa standard definition haptic signal. The basic haptic signal can also bepadded with zeros so that it contains the same number of bits as thestandard definition signal. In some embodiments, however, aconfiguration option can allow different channels to have differentnumbers of bits in each sub-sample.

At 915, information about each stream is used to define the header.Additional parameters can be specified to define other elements, such asthe target haptic output devices for each channel. At 920, each sampleof each signal is combined to create a data sample that has the samenumber of bits as each sample combined. Each sample becomes a sub-sampleor channel sample for the multi-channel stream at any given sample.

For demultiplexing, at 950 a multi-channel multiplexed stream isreceived by a device for demultiplexing. As referenced above, the devicemay be the haptic playback device or a server that extracts a stream toprovide to the haptic playback device. In some embodiments, a playbackdevice can submit its capabilities to a streaming server that selectsthe channel based on the playback device capabilities and streams onlythe selected channel to the playback device, thereby reducing thebandwidth required to play the haptic content. At 955, headerinformation for the multi-channel stream is analyzed. The number ofchannels and characteristics is determined. Based on the number of bitsand channels, the data can be parsed into individual streams. At 960,the channel to demultiplex is selected based on characteristics of thetarget haptic playback device. At 965, the haptic signal is extracted byparsing the data in the multi-channel stream. The data can be truncatedto remove added padding and down-sampled for playback on the hapticoutput device.

As disclosed, embodiments implement a haptic source signal multiplexerthat may multiplex two or more haptic signals into one multiplexedsignal. The multiplexed signal can be demultiplexed at the target deviceand the appropriate signal selected for playback on the haptic system ofthe target device. Thus, haptic designers are able to create a singlesignal with multiplexed haptic streams for playback on a variety oftarget devices having a variety of haptic output devices.

Several embodiments are specifically illustrated and/or describedherein. However, it will be appreciated that modifications andvariations of the disclosed embodiments are covered by the aboveteachings and within the purview of the appended claims withoutdeparting from the spirit and intended scope of the invention.

What is claimed is:
 1. A method of multiplexing haptic effects, themethod comprising: receiving a first haptic control signal and a secondhaptic control signal, wherein the first haptic control signal and thesecond haptic control signal are each generated from the same mediacontent, and the first haptic control signal is adapted to be renderedusing a first type of haptic output device and the second haptic controlsignal is adapted to be rendered using a second type of haptic outputdevice; combining by multiplexing the first haptic control signal andthe second haptic control signal into a single data stream, wherein eachof the first haptic control signal and the second haptic control signalcomprises a haptic data stream that comprises a respective plurality ofdata bytes, wherein the plurality of data bytes for the first controlsignal comprises discrete first samples of bits that provide informationrepresenting an on state or off state, and the plurality of data bytesfor the second control signal comprises discrete amplitude secondsamples of a signed byte of information; and sending the single datastream to a demultiplexer coupled to a playback device, wherein theplayback device comprises at least one of the first type of hapticoutput device or the second type of haptic output device.
 2. The methodof claim 1, wherein the combining comprises a bit-field combination ofthe first samples and the second samples.
 3. The method of claim 1,wherein the combining comprises substituting samples of the first hapticcontrol signal into the second haptic control signal.
 4. The method ofclaim 1, wherein the combining comprises amplitude modulating each ofthe first haptic control signal and the second haptic control signal togenerate respective amplitude modulated signals, and combining theamplitude modulated signals.
 5. A non-transitory computer readablemedium having instructions stored thereon that, when executed by aprocessor, cause the processor to multiplex haptic effects, themultiplexing comprising: receiving a first haptic control signal and asecond haptic control signal, wherein the first haptic control signaland the second haptic control signal are each generated from the samemedia content, and the first haptic control signal is adapted to berendered using a first type of haptic output device and the secondhaptic control signal is adapted to be rendered using a second type ofhaptic output device; combining by multiplexing the first haptic controlsignal and the second haptic control signal into a single data stream,wherein each of the first haptic control signal and the second hapticcontrol signal comprises a haptic data stream that comprises arespective plurality of bytes, wherein the plurality of data bytes forthe first control signal comprises discrete first samples of bits thatprovide information representing an on state or off state, and theplurality of data bytes for the second control signal comprises discreteamplitude second samples of a signed byte of information; and sendingthe single data stream to a demultiplexer coupled to a playback device,wherein the playback device comprises at least one of the first type ofhaptic output device or the second type of haptic output device.
 6. Thenon-transitory computer readable medium of claim 5, wherein thecombining comprises a bit-field combination of the first samples and thesecond samples.
 7. The non-transitory computer readable medium of claim5, wherein the combining comprises substituting samples of the firsthaptic control signal into the second haptic control signal.
 8. Thenon-transitory computer readable medium of claim 5, wherein thecombining comprises amplitude modulating each of the first hapticcontrol signal and the second haptic control signal to generaterespective amplitude modulated signals, and combining the amplitudemodulated signals.
 9. A system for multiplexing haptic effects, thesystem comprising: a receiver for receiving a first haptic controlsignal and a second haptic control signal, wherein the first hapticcontrol signal and the second haptic control signal are each generatedfrom the same media content, and the first haptic control signal isadapted to be rendered using a first type of haptic output device andthe second haptic control signal is adapted to be rendered using asecond type of haptic output device; a combiner for combining bymultiplexing the first haptic control signal and the second hapticcontrol signal into a single data stream, wherein each of the firsthaptic control signal and the second haptic control signal comprises ahaptic data stream that comprises a respective plurality of bytes,wherein the first haptic control signal comprises a full waveform havinga first frequency and having, for each sample of the first hapticcontrol signal, a value represented by a byte of information, the bytebeing one of the plurality of data bytes for the first haptic controlsignal, and wherein the plurality of data bytes for the second hapticcontrol signal comprises discrete amplitude second samples of a signedbyte of information; and a transmitter for sending the single datastream to a demultiplexer coupled to a playback device, wherein theplayback device comprises at least one of the first type of hapticoutput device or the second type of haptic output device.
 10. The systemof claim 9, wherein the combining comprises up-sampling the secondhaptic control signal to the first frequency to generate an up-sampledsecond haptic control signal and adding the first haptic control signalwith the up-sampled second haptic control signal.
 11. The system ofclaim 9, wherein the combining comprises substituting samples of thefirst haptic control signal into the second haptic control signal. 12.The system of claim 9, wherein the combining comprises amplitudemodulating each of the first haptic control signal and the second hapticcontrol signal to generate amplitude modulated signals, and combiningthe amplitude modulated signals.